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Abstract —The coordination of base stations in mobile ac¬ 
cess networks is an important approach to reduce harmful 
interference and to deliver high data rates to the users. Such 
coordination mechanisms, like Coordinated Multi-Point (CoMP) 
where multiple BSs transmit data to a user equipment, can be 
easily implemented when centralizing the data processing of the 
base stations, known as Cloud RAN. 

This centralization also imposes significant requirements on 
the backhaul network for high capacities and low latencies for 
the connections to the base stations. These requirements can 
be mitigated by (a) a flexible placement of the base station 
data processing functionality and by (b) dynamically assigning 
backhaul network resources. We show how these two techniques 
increase the feasibility of base station coordination in dense 
mobile access networks by using a heuristic algorithm. 

We furthermore present a prototype implementation of our 
approach based on software defined networking (SDN) with 
OpenDaylight and Maxinet. 

1. Introduction 

To handle the increasing wireless data rate demands of users 
in 5G mobile networks (T], it is not only important to efficiently 
utilize the available wireless resources to provide high data 
rates but also to coordinate wireless transmissions to reduce 
harmful interference. Such coordination mechanisms require 
a set of base stations (BSs) that implement a coordination 
scheme to serve one or a group of user equipments (UEs). 
We call this set of BSs a coordinated base station set (CBS). 
Hence, as the coordination mechanisms are implemented in 
the wireless domain, the selection of BSs for the CBSs has 
to be decided based on the characteristics of the wireless 
channel 0 . 0 - Implementing coordination among BSs is only 
possible if capacity and low-latency connections are provided 
by the backhaul network that interconnects the BSs. Thus, the 
constraints on the backhaul network have to be considered in 
addition to the wireless characteristics when implementing BS 
coordination mechanisms 0-©- 

In our previous work ©, we have shown how dynamic 
backhaul reconfiguration can increase the feasibility of base 
station coordination in networks with limited backhaul network 
resources, but we did not take the new challenges from very 
dense wireless access networks into account. Thus, in this paper 
we extend our dynamic backhaul reconfiguration approach to 
handle hotspot areas with a high density of users, i.e. a high 
density of desired CBS. We also show that this extension is 
beneficial for scenarios without hotspots and how it dynamically 
adapts to the density of hotspots. 


We furthermore present a prototype implementation of our 
approach that is based on OpenDaylight ©, a production- 
grade SDN controller, and Maxinet ©, a distributed emulation 
framework for OpenElow SDNs. 

Base station coordination can be implemented in different 
ways. A very promising implementation strategy is to include 
the coordination into the baseband unit (BBU) of a Cloud 
RAN. Existing work |[T0|- 0 has already shown that using 
Cloud-RAN is a good option to implement in future dense 
wireless access networks, in particular in conjunction with the 
SDN paradigm 0-0. 

Another approach to base station coordination is the idea 
to apply the software defined networking (SDN) paradigm 
also to the control and coordination of BSs. There the 
control of coordination mechanisms is also moved to a local 
controller (EC) node, thus constraints similar to CoMP between 
the controller node and the BSs also apply here. Different 
approaches for that have been proposed 0, 0. 

The required flexibility and capacity in the backhaul network 
for our approach can be provided by optical backhaul networks 
based on WDM-PON. WDM-PON 0 is an optical fiber 
technology that multiplexes optical carrier signals on an optical 
fiber using different wavelengths or colors of laser light. This 
allows great flexibility in deploying and operating backhaul 
networks: wavelengths can be added, dropped or manipulated 
in network nodes, allowing to build dynamic topologies in the 
backhaul network. This flexibility is mainly enabled by the 
fact that WDM allows operations like multiplexing different 
wavelengths and converting wavelengths in a purely optical way, 
avoiding slow and energy-consuming optical-electrical-optical 
conversion. Using optical links in a Cloud RAN backhaul has 
also been recommended by existing work 0, 0. 

The rest of the paper is structured as follows: We first 
discuss related approaches in Sectio n [H] Then we describe 
our extended algorithm in Section |lil| and our prototype 
implementation in Section |IV| including an overall system 
architecture. In Section |V] we provide both simulation results 
on the performance of our approach and on the feasibility of 
our prototype implementation. Einally, we conclude our work 
in Section 

II. Related Work 

Bartelt et al. | |20) analyze the challenges for backhaul 
networks arising from a cloud-based RAN and conclude 
that optical networks are a promising technology for Cloud 


CBS Prioritization 

[> decide for each CBS if it belongs to a hotspot 
t> output, list of hotspot CBSs and normal CBSs 
For each CBS in Whotspot, then for each CBS in Wnormai 

1) Maximum-Path BFS 

> start modified BFS from each vertex 

> output. BFS tree for each vertex 

2) Match CBS 

> match BFS trees against CBS 

> output: possible candidate BFS trees for CBS 

3) Back-Track BFS Trees 

a) Check constraints 

> recheck constraints on candidate BFS trees 

b) Wavelength Assignment 

> determine wavelengths for all candidate BFS trees 

> output: possible candidate BFS trees and their wavelength 
assignment for CBS 

4) Match CBS 

> match BFS trees against CBS again 

> output: confirmed candidate BFS trees for CBS 

5) Find Best BFS Tree 

> compare candidate BFS trees 

> output: best BFS tree for CBS 


Figure 1. Heuristic algorithm overview 

RAN backhauls, if the data rate and latency demands are 
included into a joint optimization of radio access and backhaul. 
The challenges of the backhaul connection of small cells 
in heterogeneous mobile access networks is studied in pT] |, 
resulting in a novel CBS selection algorithm to include the 
backhaul constraints. Liu et al. ]22| | investigate a dynamic 
backhaul network for the dynamic assignment of base stations 
to BBU pools with a reconfigurable backhaul architecture, 
including a small testbed evaluation. 

The issues in backhaul networks with limited resources 
together with implementing wireless coordination have been 
investigated in in existing work: Evaluations from Biermann 
et al. 0 show how a limited backhaul network reduces the 
efficiency of CoMR Soliman et al. [23 1 analyze how the 
backhaul resources have to be shared to achieve a feasible 
data exchange between coordinated BSs, but their model only 
considers two BSs. The effects of a constrained backhaul on 
different BSs coordination schemes are investigated in [24 [ , 
but without considering the selection of feasible CBSs. 

III. Heuristic Algorithm 

To deploy our algorithm in dense scenarios with hotspots 
of CBSs, we extend the heuristic algorithm from our previous 
work I?) by a CBS prioritization mechanism. This mechanism 
is explained in detail in this section together with a brief 
explanation of the whole algorithm. 

The inputs for the heuristic are the backhaul network as an 
annotated graph G = (V, S), a set of available wavelengths per 
link K and the desired CBS W with each Wi gV together 
with their constraints on data rate and latency. 

An overview of the heuristic is depicted in Figure and 
the individual steps are explained below. 

CBS Prioritization: This new step of the algorithm divides 
the desired CBS into two sets Whotspot and Wnormai- The 


calculations in this step use two threshold values ty, which 
determines how strict the filtering of hotspot vertices should 
be, and which determines how strict the filtering of hotspot 
CBS should be. The numbers for the threshold values depend 
on the scenario and can be determined using a parameter study. 

The algorithm first calculates for each vertex v from the 
backhaul graph in how many CBS the vertex is present as: 

- ^VtzWi 

Ijv denotes an indicator variable with value 1 if condition X 
is true and value 0 otherwise. Each vertex v is considered a 
hotspot vertex if 

hy > \W\ ■ ty 

and is added to the set of hotspot vertices 14. Now each CBS 
Wi is added to Whotspot if 

— TO — - ^ 

otherwise it is added to Wnormai- 

Now the actual algorithm for BBU/LC placement and 
backhaul resource allocation has to be executed for each CBS. 
The following steps are executed per CBS, starting with the 
CBSs from Whotspot and the CBSs from Wnormai: 

1) Maximum-Path BFS: The heuristic performs a modified 
breadth-first search from each vertex of the graph as the start 
node. Whenever a new tree edge (it, v) is discovered, the 
constraints for the new edge are checked in the following way: 

• Does the latency to the new vertex v via the whole 
path from the root via u and the new edge (it, v) not 
exceed the maximum round-trip latency? 

• Are there any wavelengths on (it, v) with enough free 
capacity to connect v to the root vertex, if v is part of 
the CBS? 

If these checks are not successful, the new vertex v is discarded, 
otherwise it is added to the tree. The result of this step is a 
set T of BFS trees, which only contain vertices that meet the 
constraints within the tree. This does not yet take into account 
reciprocal effects between multiple BSs in the CBS. 

2 ) Match CBS: The heuristic checks for every BFS tree Ti 
from step 2), whether it contains all BSs from the desired CBS. 
The result is a set of only these BFS trees T that match the 
desired CBS. 

3) Back-Track BFS Trees: In the constraint checking in step 
F) vertices were discarded based on the latency for the full 
paths to the start node and the link capacity for only single 
links and not the full paths. Here the capacity constraints are 
checked again taking into account the capacities on full paths. 

If the constraints are not violated a wavelength assignment 
for all routing paths between the BSs from the CBS and the 
BFS tree root has to be determined. Because the required 
data structures for this step can easily be constructed in the 
back-tracking phase, the back-tracking and the wavelength 
assignment are executed in one step. 

Details on the implementation of the wavelength assignment 
can be found in our previous work ||7). 





4) Match CBS: After removing vertices in the previous step, 
the data on trees matching the CBS is not valid anymore. Thus, 
step 2j is repeated. If a tree still matches the CBS, the starting 
node is a valid candidate for a controller location for that CBS. 

5) Find Best BFS Tree: Finally, the best BFS tree from 
the remaining candidates has to be determined. The algorithm 
calculates three different costs per candidate tree and sums 
them up in a weighted sum n per tree 

n = '^{Wg -ng + Wa-Ua + wi- ni) 

with 

• Ug as the total number of wavelengths used in the tree 

• Ua as the number of wavelengths that have to assigned 
additionally for using that tree 

• ni as the number of used links 

and Wg, Wa and wi as weight factors. 

The algorithm then selects the tree with the lowest total 
cost, sets the root BS as the controller, stores the routing paths 
and updates the annotations on G for running the next iteration 
for the next CBS. 


IV. Prototype 

In this section we first describe the architecmre of a real- 
world implementation of our approach and then introduce how 
we implemented our prototype. 

A. System Architecture 

Our system architecture is based on the decoupling of the 
following three components of the overall system; 

• Application 

• Controller 

• Network 

The architecture of our system is shown in Figure 

The application is essentially the algorithm from Section 
[III| together with necessary data structures to store the input 
and output data at runtime. The application is decoupled 
from the controller by accessing two different APIs of the 
controller. With the OpenFlow API the application can both 
query the topology and flow configuration of the backhaul 
network and modify the flow configuration according to the 
algorithm outputs. In addition to this API, the application also 
needs to access the BBU/LC API in order to query the status 
of running BBUs/LCs and to start or stop BBUs/LCs. 

The controller acts as the centralized link between the 
application and the backhaul network, like the controller in 
an SDN network GD- It exposes the previously described 
northbound APIs to the application and controls both the 
backhaul network and the BBU/LC instances via its southbound 
plugins. Consistent with the northbound APIs the controller 
needs both an OpenFlow plugin and a BBU/LC control plugin. 

Of course the backhaul network has to be based on 
OpenFlow-enabled hardware, i.e. switches, otherwise the con¬ 
troller could not use the OpenFlow plugin to reconfigure 
the backhaul network. Also all potential nodes for hosting 
a BBU/LC have to run a hypervisor to allow the dynamic 
instantiation of BBU/LC instances. 


Figure 2. 




B. Implementation 

Our reference implementation of the described system 
architecture is based on the OpenDaylight controller platform 
1^. OpenDaylight already includes a fully featured OpenFlow 
plugin, thus no additional implementation is required for 
this part. Also the model-based design behind OpenDaylight 
facilitated the implementation of a BBU/LC control plugin. 
A core concept with OpenDaylight is the implementation of 
REST-based northbound APIs. The OpenFlow plugin already 
provides APIs for both querying the topology and configuring 
flows within the backhaul network. For our BBU/LC control 
plugin, we implemented REST APIs accordingly. 

Our implementation of the application is based on Python 
and solely relies on the REST APIs of the controller. A small 
wrapper converts data between the OpenDaylight API format 
and the data structures for our algorithm. This wrapper is 
tailored to the OpenDaylight API format, but it could in 
principle be adapted to any other OpenFlow controller, making 
our algorithm implementation independent of the controller 
platform. 

C. Testbed 

Because we cannot test our application and controller on a 
real-world OpenFlow enabled backhaul network, we have to use 
an emulated network to test and evaluate our implementation. 
For this we use Maxinet an extension to the well known 
Mininet emulator | |25] | for distributed emulation, to emulate 
a fully functional, virtual OpenFlow enabled network on a 
cluster of physical machines. We use a small setup with four 
physical machines, as shown in Figure]^ One machine hosts the 
application, OpenDaylight and the Maxinet frontend, the three 
other machines are used as Maxinet workers to emulate the 
backhaul network. Since we do not have any wireless interface 
integrated into the testbed, the traffic between the BSs and the 
BBUs/LCs is emulated using iperf p6). 



























Figure 3. Testbed 


V. Evaluation 

In this section we present (a) simulation results that show 
how our new algorithm performs compared to the algorithm 
from our previous work ||7) in Section |V-A| And (b) a testbed 
evaluation to show how our prototype implementation works 
in Section IV-BI 


A. CBS Prioritization Simulation 

1) Simulation Scenario: A fixed number of BSs are placed 
on a regular grid, with a mean inter-BS distance of s = 1000 m 
(urban scenario 0), and are then shifted in both x and y 
direction according to two independent, normally distributed 
random variables with zero mean and standard deviation |. 


To determine the threshold values for the CBS prioritization 
tv and th, we performed a parameter study and identified 
ty = 0.1 and tfi = 0.9 as the best values to maximize the 
number of feasible CBSs in this scenario. 

2) Simulation Results: The results for the simulation are 
shown in Figure where we investigate the influence of 
different hotspot CBS fractions h. For /i = 0 no hotspot CBSs 
exist and for h = 1 all CBSs are hotspot CBSs. Solid lines 
are our new algorithm with the CBS prioritization step, dashed 
lines the algorithm from our previous work as a reference. 

In Figures to we show the resulting feasibility of the 
CBS, i.e. the fraction of desired CBSs that were successfully 
established. Even for a scenario with no hotspots {h = 0) the 
CBS prioritization step increases the CBS feasibility to 100% 
for all capacity demands and all numbers of desired CBSs. 
In the scenarios with h = 0.5 and h = 0.75, the feasibility 
drops as the number of desired CBSs increases. Still the CBS 
prioritization step increases the CBS feasibility by over 20%. 
In the scenario with all desired CBSs in the hotspot {h = 1), 
there is no significant improvement from prioritizing CBSs, 
which is the expected outcome. 

Figures to show the results for the overall number 
of used wavelengths and Figures to show the results for 
the number of used wavelengths per link. In the scenario with 
no hotspot CBSs {h = 0) the CBS prioritization significantly 
improves the efficient use of wavelengths, as both the total 
number of used wavelengths and the number of wavelengths per 
link are significantly lower when using the CBS prioritization. 
This effect is also evident in the scenarios with h = 0.5 and 
h = 0.75. For h = 1 there is again no signihcant difference. 

B. Prototype Evaluation 


The backhaul network is created as a mesh topology where 
two BSs are connected by a link if their distance is less than 
or equal to 1.5 • s. This value produces a partially connected 
mesh network; smaller or larger values result in too sparse 
or too dense topologies, which are not realistic. All links in 
the backhaul network are assigned the same set of available 
wavelengths AT = 4 and each wavelength is assigned the 
same fixed capacity of 2.5 Gb/s. The latency for each link 
is determined by the distance multiplied by 1.45 divided by the 
speed of light as we are modeling an optical backhaul network. 

In order to create a hotspot of CBSs, a x,y coordinate is 
selected uniformly as the hotspot center. Now a fraction h 
of all CBSs is placed around the hotspot center based on a 
normal distribution with the hotspot coordinate as the mean 
and standard deviation | as the desired hotspot CBSs. All 
other desired CBSs are generated by placing then uniformly 
at random on the plane covered by the placed BSs. The BSs 
which are considered as the CBS are all BSs located inside a 
circle around the coordinates of the CBS with a given radius. 
We determine this radius by multiplying the mean inter-BS 
distance with a factor r = 1.5, which results In 5 BSs per CBS 
on average. 

The capacity demand d for each BS in the CBS is set to the 
same value and is either 0.625 Gb/s, 1.25 Gb/s or 2.5 Gb/s. This 
implies that at a demand of 2.5 Gb/s one complete wavelength 
is required to connect a BS to the controller. 


The prototype evaluation uses our testbed implementation 
described in Section IIV-CI 

1) Prototype Scenario: As a scenario for the prototype 
evaluation we use a smaller scenario with only 16 BSs. To 
avoid a bottleneck from the Maxinet worker interconnect, a 
CBS can only contain BSs being emulated on the same worker 
machine. Apart from this limitation, CBSs are generated in 
the same random way as in the simulation (Section |V-A| l. All 
emulated links also have a capacity of 2.5 Gb/s, and for the 
demand per BS we only consider 1.25 Gb/s, because the other 
values from the simulation (0.625 Gb/s and 2.5 Gb/s) do not 
provide additional insights from the evaluation results. 

We also use three different implementations for the appli¬ 
cation to compare the performance of our algorithm: 


Full Backhaul Reconfiguration uses our full algorithm 
with the full flexibility in 
rk conflguration and BBU/FC 


described in Section III 


terms of backhaul network 
placement. 

Static Backhaul Reconfiguration uses a limited version 
of our algorithm, where the BBU/FC is placed on a 
fixed BS and the algorithm only performs the flow 
routing and wavelength assignment. 

Static Backhaul does not use our algorithm at all 
and relies on a fixed BBU/LC placement, a static 
assignment of wavelengths and always uses shortest 
paths for the flow routing. 
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Figure 4. CBS prioritization simulation 


2 ) Prototype Results: For the prototype evaluation, we 
consider three different metrics. 

The a priori feasibility is calculated from the output of the 
application. In Figure]^ we can see that the a priori feasibility 
decreases for both applications that are based on our algorithm 
as the number of desired CBSs increases. This is due to the 
limited available resources in the backhaul network. The results 
also show that using the full flexibility of our algorithm yields 
a better a priori feasibility than using the limited version. The 
a priori feasibility for the static backhaul implementation is 
always 100%, because this implementation does not consider 
any resource constraints in an a priori way. 

In order to measure the a posteri feasibility of all imple¬ 
mentations we perform a UDP throughput measurement using 
iperf ]26| with a desired throughput of 1.25 Gb/s. As results 
from this measurement we obtain both the achieved throughput 
(Figure [5bjl and the packet loss (Figure [5^. We can see that 
both implementations with our algorithm achieve the desired 
throughput and do not cause any packet loss. In contrast to 
that the static implementation is not able to achieve the desired 
throughput and causes a signihcant packet loss. 


VI. Conclusion 

We have presented an extended approach to verify the 
feasibility of base station coordination in a centralized RAN 
environment considering capacity and latency constraints with 
a backhaul network with limited resources. This approach 
includes the dynamic assignment of backhaul resources, which 
we call backhaul network configuration, as well as the dynamic 
instantiation of BBUs or local controllers (LCs). 

Our simulation shows that our extension enables the use of 
our approach in dense wireless access networks with hotspots of 
users, and furthermore increases the feasibility of base station 
coordination in networks without hotspots. This is a signihcant 
improvement compared to our previous work |j^. 

We have also presented a prototype implementation for a 
real-world deployment of our approach in a testbed. Our testbed 
measurements show that our approach is able to guarantee the 
availability of the desired resources for each feasible CBSs, and 
increase the feasibility of CBSs compared to a static assignment 
of backhaul resources. 
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Figure 5. Prototype evaluation 
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